Data center network

ABSTRACT

The present disclosure provides a data center network having one or more data center rows, where each row has one or more racks, and each rack has one or more network devices, such as servers, storage devices and switches. The rows and racks are interconnected by a fiber interconnect core that reduces the number of switching nodes in the data center network, and reduces the individual path latency, the overall data center network cost, power consumption, and power and cooling requirements.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority U.S. Provisional Application No.62/057,086, filed on Sep. 29, 2014, entitled “Data Center Network,” andU.S. Provisional Application No. 62/057,008, filed on Sep. 29, 2014,entitled “System for Increasing Fiber Port Density in Data CenterApplications” both of which are incorporated herein in their entirety byreference.

BACKGROUND ART

Field

The present application relates generally to communication networks, andmore particularly to data center networks with improved interconnectionsand improved interconnection management.

Description of the Related Art

Communication networks have a long history, evolving from singletransmission lines and manual switching, to early multi-line automaticelectro-mechanical switching systems, to more recent electronic andoptical transmissions across many lines or fibers using electronic oroptical switching systems.

Today's digital and optical switching systems allow for substantialgrowth in the size of communication networks to meet the needs of everexpanding communication networks. The progression to the more commondigital and optical switching systems was spurred on a belief that newersemiconductor (e.g., VLSI) and optical devices met the need for highspeed data transmissions.

With the evolution of communication switching systems has been theevolution of computers and the information age. In order to manage theincrease in data transmissions between computers, data centers came tobe. Data centers have their roots in the huge computer rooms builtduring the early ages of the computing industry. Early computer systemswere complex to operate and maintain, and required a special environmentin which to operate. During the boom of the microcomputer industry inthe 1980s, computers started to be deployed everywhere and systems, suchas dedicated computers or servers, were developed to meet the demandscreated by the need to have the increasing number of computerscommunicate. During the latter part of the 20^(th) century and earlypart of the 21^(st) century, data centers grew significantly to meet theneeds of the Internet Age. To maintain business continuity and growrevenue, companies needed fast Internet connectivity and nonstopoperations to establish a presence on the Internet.

Today, data centers are built within the enterprise network, a serviceprovider network, or a shared, colocation facility where the networks ofmany disparate owners reside. With the significant increase in businessand individual use of the Internet, and the significant need forbandwidth to transmit high volumes of data, especially video andgraphics, data centers are again under pressure to evolve to handle theboom in growth. However, data centers are typically very expensive tobuild, operate and maintain, and data center operators are searching forways to reduce costs while increasing data processing and transmissioncapabilities, while meeting all reliability requirements.

In order to meet the increased demands, data center networkarchitectures have changed. Sometimes the changes to the networkarchitecture require significant rerouting of network connections, andsometimes the network architecture needs to be dynamic, changingfrequently. And, all this has to be achieved at today's fast rates withlittle or no failures or delays in the transmission of data.

One area where the data center network is changing is with networkswitches that have evolved with the capability of switching data trafficon a packet-by-packet basis, which is known as packet switching. Whilepacket switching can change the physical route of individual packetsthrough a network, there are some network applications where therequirement is to switch all the data traffic from one physical route toa second physical route through the network, which is known as portswitching or path switching.

Traditionally, data center network devices, such as servers, storagedevices, switches, and routers, as well as NIC cards that may be addedto such devices have physical connection points to transmit and receivedata. These connection points generally include a transceiver and aconnector, which are often referred to as a port. Ports can be copper orfiber ports that are built into the device, or the ports can be plug-inmodules that contain the transceiver and connector and that plug intoSmall Form Factor (SFF) cages intended to accept the plug-intransceiver/connector module, such as SFP, SFP+, QSFP, CFP, CXP, andother transceiver/connector modules, where the connector extends from anexterior surface of the device, e.g., from a front panel. Fiber portsmay be low density or single fiber ports, such as FC, SC, ST, LC, or thefiber ports may be higher density MPO, MXC, or other high density fiberports.

Fiber optic cabling with the low density FC, SC, ST, or LC connectors orwith SFP, SFP+, QSFP, CFP, CXP or other modules either connect directlyto the data center network devices, or they pass through interconnectorcross connect patch panels before getting to the data center networkdevices. The cross connect patch panels have equivalent low density FC,SC, ST, or LC connectors, and may aggregate individual fiber strandsinto high density MPO, MXC or other connectors that are primarilyintended to reduce the quantity of smaller cables run to alternatepanels or locations.

From a logical perspective, traditional data center networks, as shownin FIG. 1, includes of servers 104 and storage devices 106, plusconnections between the servers, storage devices and to externalinterfaces. A data center interconnects these devices by means of aswitching topology implemented by pathway controlling devices 130, suchas switches and routers. As networks grow in size, so does thecomplexity. The servers 104 and storage devices 106 connect to oneanother via cable interfaces 118, 120, 122, and 124. Interconnects 112are used to bundle and reconfigure cable connections between endpointsin cable bundles 114, 116, and 126. The Management Controller 100configures and controls and receives status information from the datacenter network devices via management interface path 101. As can be seenin FIG. 1, data center networks become layered with multiple pathwaycontrolling devices 130 in an attempt for every endpoint to have thecapability of switching and/or routing data packets to any otherendpoint within the data center network. This can result in very complexhierarchical switching networks which in turn require considerable powerand expense in order to maintain and respond to configuration changeswithin the network.

From a physical perspective, a typical data center networkconfiguration, shown in FIG. 2, includes multiple rows of cabinets,where each cabinet encloses a rack of one or more network devices, e.g.,switches 102, servers 104 and storage devices 106. Typically, for eachrack there is a top-of-rack (TOR) switch 102 that consolidates datapacket traffic in the rack from each server 104 and storage 106 viacables 140 and transports the data packet traffic to a switch known asan end-of-row (EOR) switch 108 via cables (not shown). The EOR switch istypically larger than a TOR switch, and it processes data packets andswitches or routes the data packets to a final destination or to a nextstage in the data center network, which in turn may process the datapackets for transmission outside the data center network. Typically,there are two TOR switches 102 for every rack in a row, e.g. Rows 1 and2, and two EOR switches 108 for each row, where the second switch ineach case is typically for redundancy purposes.

In one configuration, a TOR switch 102 will switch data packet trafficdirectly between any two network devices, e.g., servers 104 or storagedevices 106, within a given rack. Any data packet traffic destined forlocations outside of the rack is sent to the EOR switch 108. The EORswitch 108 will send data packet traffic destined for a network devicein a different rack in the same row to the TOR switch 102 of the rackwhere the network device resides. The TOR switch 102 within thedestination rack will then forward the data packet traffic to theintended network device, i.e., the destination device. If the datapacket traffic is for network devices outside of the row, e.g., Row 1,the EOR switch 108 will forward the traffic to core switch 110 forfurther transmission.

In other configurations, a TOR switch 102 may be used as an aggregator,where all data packet traffic is collected and forwarded to an EORswitch 108. The EOR switch then determines the location of thedestination network device, and routes the data packet traffic back tothe same TOR switch 102 if the data packet traffic is destined for anetwork device in that rack, to a different TOR switch 102 in adifferent rack if the traffic is destined for a network device in adifferent rack in the same row, or to the core switch 110 if thedestination of the data packet traffic is outside of that row.

The TOR switch 102 may couple the entire data packet traffic from aningress port to an egress port, or may selectively select individualpackets to send to an egress port. Referring to FIG. 3, in conventionalapplications, a TOR switch 102 retrieves header information of anincoming data packets on an ingress port of the TOR switch, and thenperforms Access Control List (ACL) functions to determine if a packethas permission to pass through the TOR switch 102. Next, a check is runto see if a connection path was previously based on the information fromwithin the packet header. If not, then TOR switch 102 may run OpenShortest Path First (OSPF), Border Gateway Protocol (BGP), RoutingInformation Protocol (RIP), or other algorithms to determine if thedestination port is reachable by the TOR switch 102. If the TOR switch102 cannot create a route to the destination network device, the packetis dropped. If the destination network device is reachable, the TORswitch 102 creates a new table entry with the egress port number,corresponding egress header information, and forwards the data packet tothe egress port. Using this methodology, the TOR switch 102 transfers,or switches, the data packet from the ingress port to the requiredegress port.

Traditional data center architectures have not had the capability to mapout the physical interconnections between pathway controlling devices130, servers 104, storage devices 106, and other devices in the datacenter network. Existing network applications, such as AddressResolution Protocol (ARP), Spanning Tree, OSPF and others, map outlogical interconnections between two devices connected together, butsuch network applications do not provide information about the physicalinterconnections. As a result, in the event of a link failure, the enddevices are aware of the failure, but cannot identify the physicalinterconnection which requires repair.

BRIEF SUMMARY

The present disclosure provides a data center network comprising one ormore rows, wherein each row has one or more racks, and wherein each ofthe one or more racks has at least one network device and at least onetop-of-rack network switch, and at least one end-of-row fiber meshinterconnect in communication with each top-of-rack network switchwithin the same row of the one or more rows, such that each top-of-racknetwork switch has a direct connection to every other top-of-racknetwork switch within the same row. In an exemplary embodiment, eachtop-of-rack network switch comprises a housing having one or moreconnection panels, and a set of ports, wherein each port within the setof ports is configured to receive data streams from at least one networkdevice within each of the one or more racks, and to transmit datastreams to at least one network device within each of the one or moreracks, wherein each port in the set of ports includes a connector and atleast one transceiver optically coupled to the connector, and whereinthe connector is mounted to the one or more connection panels forconnecting to the at least one network device and the end-of-row fibermesh interconnect.

The present disclosure also provides a data center network, comprisingone or more rows, wherein each row has one or more racks, and whereineach of the one or more racks has at least one network device and atleast one top-of-rack fiber mesh interconnect, and at least oneend-of-row fiber mesh aggregation in communication with each top-of-rackfiber mesh interconnect within the same row of the one or more rows,such that each top-of-rack fiber mesh interconnect has a directconnection to every other top-of-rack fiber mesh interconnect within thesame row. In an exemplary embodiment, each top-of-rack fiber meshinterconnect comprises a housing having one or more connection panels,wherein each connection panel includes a plurality of connectors, and aplurality of optical fibers within the housing and connected between oneor more of the plurality of connectors in a predefined mapping toprovide a direct optical fiber connection between connectors. In anexemplary embodiment, each end-of-row fiber mesh aggregation comprises ahousing having one or more connection panels, wherein each connectionpanel includes a plurality of connectors, and a plurality of opticalfibers within the housing and connected between one or more of theplurality of connectors in a predefined mapping to provide a directoptical fiber connection between connectors.

The present disclosure also provides a data center network fiber meshinterconnect device. The, fiber mesh interconnect device may comprise ahousing having one or more connection panels, wherein each connectionpanel includes a plurality of connectors, and a plurality of opticalfibers within the housing and connected between one or more of theplurality of connectors in a predefined mapping to provide a directoptical fiber connection between connectors.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a conventional logical data center networktopology;

FIG. 2 is a block diagram of a row architecture in a conventional datacenter network;

FIG. 3 is a flow diagram for a top of rack switch in a conventional datacenter network;

FIG. 4 is a block diagram of an embodiment of a row architecture in adata center network according to the present disclosure;

FIG. 5 is a block diagram of an embodiment of the interconnectionbetween high density racks in the data center network according to thepresent disclosure;

FIG. 6 is a block diagram of another embodiment of a row architecture ina data center network according to the present disclosure;

FIG. 7 is a block diagram of another embodiment of a row architecture ina data center network according to the present disclosure;

FIG. 8 is a block diagram of another embodiment of a row architecture ina data center network according to the present disclosure;

FIG. 9 is a block diagram of embodiments of a fiber mesh interconnectarchitecture according to the present disclosure;

FIG. 10 is block diagram of another embodiment of a fiber meshinterconnect architecture according to the present disclosure;

FIG. 11 is block diagram of another embodiment of a fiber meshinterconnect architecture according to the present disclosure;

FIGS. 12-17 illustrate an embodiment of physical fiber meshinterconnection architecture according to the present disclosure;

FIG. 18 is block diagram of another embodiment of a fiber meshinterconnect architecture according to the present disclosure;

FIG. 19 is block diagram of another embodiment of a fiber meshinterconnect architecture according to the present disclosure;

FIG. 20 is a block diagram of an embodiment of a NIC card architectureaccording to the present disclosure;

FIG. 21 is a block diagram of a data center network topology accordingto the present disclosure;

FIG. 22 is a block diagram of another embodiment of a NIC cardarchitecture according to the present disclosure;

FIG. 23 is a block diagram of an embodiment of a high density pathwaycontrolling device architecture according to the present application;

FIG. 24 is a block diagram of another embodiment of a high densitypathway controlling device architecture according to the presentapplication;

FIG. 25 is a block diagram of an embodiment of the architecture of theorchestration system according to the present disclosure;

FIG. 26 is a flow diagram for a cable verification process according tothe present application;

FIG. 27 is a flow diagram for an embodiment of a discovery processaccording to the present application;

FIG. 28 is a block diagram of a data center network topology accordingto the present disclosure;

FIG. 29 is a flow diagram for determining connectivity paths withconstraints in accordance with one embodiment of the present disclosure;and

FIG. 30a is a block diagram of a data center network topology accordingto the present disclosure with identifiable connections;

FIG. 30b is a block diagram of a data center network according to thepresent disclosure with unidentifiable connections at the edges;

FIG. 30c is a block diagram of a data center network topology accordingto the present disclosure with unidentifiable connections in thenetwork;

FIG. 31 is a block diagram of another embodiment of data center networktopology according to the present disclosure; and

FIG. 32 is a flow diagram for detecting broken data paths in accordancewith one embodiment of the present disclosure.

DETAILED DESCRIPTION

In this disclosure, a connection can be a single copper or fiberconnection or a duplex connection having a transmit connection and areceive connection. For ease of drafting, reference to a connection orconnections includes both a single connection or a duplex connection.

The data center network of the present disclosure provides a new classof high port density network switches. An example of a high density portnetwork switch is provided in the description in U.S. Provisional PatentApplication entitled “System For Increasing Fiber Port Density In DataCenter Applications”, Ser. No. 62/057,008, filed Sep. 29, 2014, which isincorporated herein in its entirety by reference. Utilizing the highport density network switch elevates the Top of Rack (TOR) switches 102to High Density Top of Rack (HD TOR) 202 switches, and along with newfiber interconnection methodologies, can be configured as aninterconnection fabric, replacing or significantly reducing the need forEnd of Row (EOR) switches 108 and in some cases core switches 110.

The data center network of the present disclosure creates a switchapplication including High Density Top of Rack switches 202 with directconnection of dedicated bandwidth to every other HD TOR switches 202within a row and utilizing a new End of Row Fiber Interconnect Mesh 204application. The End of Row Fiber Interconnect Mesh 204 comprises afiber interconnect scheme containing prewired fiber connectionsconfigured for a particular data center row application and alsoprovides multiple routes to other racks within the row, as well asconnectivity to other rows and to the core.

The overall physical network is managed by a Fiber Interconnect Meshorchestration system 400 which can learn the logical and physical datacenter network topology, and can define paths through theinterconnection fabric to provide efficient connections betweenendpoints. The HD TOR Switches 202 and EOR Aggregation 210 are differentfrom conventional TOR Switches 102 and EOR Switches 108 in that they aredesigned to function with the End of Row Fiber Interconnect Mesh 204 andTop of Rack Fiber Mesh Interconnect 208. One embodiment of a networkconfiguration implementing the present disclosure uses the fiber meshinterconnects 10 shown in FIGS. 12-17 for a 12 rack implementation whereFIG. 12 is a Top of Rack Fiber Mesh Interconnect 208 which provides theconnectivity for the individual top of rack interconnections, FIG. 13and FIG. 14 show two fiber mesh interconnects 10 which create the End ofRow Fiber Mesh Aggregation 210 implementation. FIG. 13 shows the End ofRow fiber interconnections for connections to End of Row Fiber MeshAggregation modules 210 to other rows in the data center network, to thecore switches 110, and to connections outside the network 134. FIG. 14shows the End of Row-Row Return function where connections from one rackare looped back within the same row to Top of Rack Fiber MeshInterconnects 208 or HD TOR Switches 202 in other rows. FIGS. 15-17detail the fiber mapping for this embodiment of the Row Return function.

For traffic which has known destinations, when the end destination isknown and reachable within a local environment, conventional pathwaycontrolling devices 130 used to transmit data between two endpointswithin a local region can be eliminated and replaced with direct cableconnections. By physically connecting predefined traffic directly fromone endpoint to another, the complexity of the network is reduced due tothe reduction in the number of pathway controlling devices 130.Accordingly, the cost associated with conventional pathway controllingdevices 130 is eliminated, the power consumption associated with thesepathway controlling devices 130 is eliminated, the heat dissipationassociated with these pathway controlling devices 130 is eliminated, andthe real estate requirements in the data center associated with thesepathway controlling devices 130 is significantly reduced and replaced bycables and or interconnect panels.

Referring now to FIG. 4, an embodiment of the data center networkarchitecture according to the present application is shown. In thisembodiment traditional TOR Switches 102 are replaced by a High DensityTop of Rack (HD TOR) Switches 202 and the EOR switch 108 is replaced bywith an End of Row Fiber Mesh Interconnect 204. In this configuration,the HD TOR Switch 202 has sufficient ports to connect to each of theother racks within the row or to the core switch 110 by interconnectionsestablished by the End of Row Fiber Mesh Interconnect 204.

In the embodiment of the FIG. 4 row 1 has six racks with a HD TOR switch202. FIG. 5 shows an embodiment of two of the racks in a row, where thetwo racks are coupled together to create a double wide rack 206configured as an odd and even rack. In this embodiment, ports from HDTOR Switch 202 connect to the servers 104 and/or storage devices 106 inan even and an odd rack, e.g., Rack 1 and Rack 2. In one implementationof this embodiment, a 128 port HD TOR switch 202 in each double widerack 206 can provide 42 ports to network devices (e.g., servers 104,storage devices 106, and other network devices) contained in the oddrack (Rack 1) and 42 ports to network devices (e.g., servers 104,storage devices 106, and other network devices) contained in the evenrack (Rack 2) for a total of 84 ports to the double wide rack 206. Inthis exemplary embodiment, each HD TOR switch 202 can have 4 portsconnected to each of the other adjacent double racks, such that in a 12rack row configuration a total of 20 ports are allocated. This leaves 24ports to the EOR fiber core, e.g., fiber mesh interconnect 204. Theactual implementation arrangement may be different for different datacenter configurations depending upon for example the size of the datacenter, type of traffic, the traffic models, requirements for interserver communications, and other attributes. As the number of ports onthe HD TOR switches increases, this embodiment can support more portsper device, more devices per rack, or more racks per HD TOR Switch 202.

An alternate embodiment may include a double height single rackconfiguration in locations where vertical height for taller racks is nota concern.

The embodiment in FIG. 6 shows a different implementation whereconventional End of Row Switches 108 are replaced with End of RowAggregation 210, and where conventional Top of Rack Switches 102 arereplaced by TOR Fiber Mesh Interconnects 208. The TOR Fiber MeshInterconnects 208 are passive optical interconnects that employ a fibermesh structure (an example is seen in FIGS. 12-17) to connect all thenetwork devices (e.g., servers 104, storage devices 106, and othernetwork devices) in a rack to the End of Row Aggregation 210, thuseliminating the need for more costly Top of Rack Switches 102. The Endof Row Aggregation 210 is different from the End of Row Switches 108 inthat they are designed to function with the Fiber Mesh Interconnects 10.

The embodiment in FIG. 7 shows a data center network that is similar tothe embodiment of FIG. 6. In this embodiment, the Core Switches 110 arereplaced by providing interconnections from the End of Row Aggregation210 in one row to End of Row Aggregations 210 in other rows. Similarly,in the configuration in FIG. 4 the Core Switches 110 can be eliminatedand the EOR fiber mesh interconnects 204 can provide theinterconnections.

Referring to FIG. 8, The EOR Aggregation 210 can be implemented in anumber of different configurations depending upon particular data centernetwork architecture requirements. In certain embodiments, at least someof the fibers from each TOR Fiber Mesh Interconnect 208 are looped backin the EOR Fiber Mesh Aggregation 212 to other TOR Fiber MeshInterconnects 208 in other racks, and to different fiber locations onthe originating TOR Fiber Mesh Interconnect 208. This permits directconnections from one rack network device to connect directly withanother network device located in the same or different rack and avoidthe latency associated with being switched by a conventional TOR switchor EOR switch.

In another embodiment, the connections are fixed and the EOR Aggregation210 may include EOR Fiber Mesh Aggregation 212. In this embodiment,fibers from the TOR Fiber Mesh Interconnects 208 would be looped to adestination either within the rack fiber interconnections or toconnections outside the data center via connection 134 either to anotherrow, or to a core switch for further switching. This also allows a coreswitch to provide switching functions if needed to selectively switchpackets or paths back into the same row without the need for switchingwithin the row.

In another embodiment, some of the fibers from the EOR Fiber MeshAggregation 212 may be fed to an End of Row Packet Switch 214 whichwould switch the individual packets based upon packet header destinationinformation and based upon instructions from the orchestration system400 which determines if the packets are to be sent back into the EORFiber Mesh Aggregation 212 for delivery to a device connected to a TORFiber Mesh Interconnect 208, or to an end location located outside theinterconnections of the rack.

In another embodiment, some of the fibers from the TOR Fiber MeshInterconnects 208 may be fed to an End of Row Path Switch 216 whichwould switch the entire optical signal from an input fiber to one ormore outgoing fibers based upon instructions from the orchestrationsystem 400. The optical path is then connected by the End of Row PathSwitch 216 to an EOR Fiber Mesh Aggregation 212 or to an end locationlocated outside the interconnections of the rack. The advantage of usingpath switches over packet switches is that a path switch hassignificantly less latency in the path because the entire path isswitched and the circuitry inside the path switch does not look at theheaders of each packet to make a decision as to where to switch thetraffic. The advantage of using a packet switch over a path switch isthat packet switches look at the headers of each packet to make adecision as to where to switch the data packet traffic and can switchindividual packets to different destinations.

In another embodiment, some of the fibers from the TOR Fiber MeshInterconnect 208 or EOR Fiber Mesh Interconnect 204 or EOR Fiber MeshAggregation 212 may be fed to an End of Row Packet Switch 214 whileothers are fed to an End of Row Path Switch 216. This permits theflexibility of packet switching for some connections as well as pathswitching for other connections under the configuration of theorchestration system 400.

An alternate embodiment for any of the previously mentioned or otherconfigurations may include a middle rack for concentration of the fiberinterconnections.

Referring to FIG. 9, each of the TOR Fiber Mesh Interconnect 208, End ofRow Fiber Mesh Interconnect 204, and EOR Fiber Mesh Aggregation 212 canalso be referred to herein as a Fiber Mesh Interconnect 10. A Fiber MeshInterconnect 10 is a system that simplifies the interconnection of fibercabling within a data center network by increasing the fiber densitywithin a small footprint. In one embodiment, the Fiber Mesh Interconnect10 includes a plurality of individual fiber strands on one or more thinfilms, such as a Mylar sheet or other suitable medium, and a pluralityof connectors 610 connected to one or more of the individual fiberstrands. The Fiber Mesh Interconnect 10 may then be installed in ahousing or enclosure to protect the fibers, as seen in FIGS. 10 and 19.Thus, the Fiber Mesh Interconnect 10 connects individual optical fiberstrands from one port to a different port within the Fiber MeshInterconnect 10.

Continuing to refer to FIG. 9, this embodiment uses a Fiber MeshInterconnect 10 where bare or coated single-mode or multi-mode fibersare placed on a thin film surface 630, such as a Mylar sheet, in orderto tightly control the route each fiber will take within the enclosure.The fibers are placed and then adhered to a thin film 630. The fiberscan be in a single layer or can be overlapping previously laid fibersthus creating a multi-layer Fiber Mesh Interconnect 10. The connectors610 are installed on to the fibers and then polished using standardfiber termination processes, or spliced to the fibers by fusion splicingor by another suitable method for terminating fibers to connectors. TheFiber Mesh Interconnect 10 is then placed in a housing or enclosure, asshown in FIG. 10. This architecture ensures each connection path withinthe Fiber Mesh Interconnect 10 is defined and routed in accordance tothe intended routing path for that Fiber Mesh Interconnect 10application.

One of the issues with using individual fiber cables with connectors isthat the cables are be placed inside the enclosure in such a manner thatthe cables do not fold or bend below the minimum bend radius recommendedfor that fiber type. Bending a fiber cable below its minimum bend radiusresults in optical power loss and potentially signal loss. This presentdisclosure contemplates adhering fibers to a horizontal plane, e.g., thethin film 630, from one connector position to another connector positionsuch that the route and the fiber bend radius is tightly controlled thusminimizing optical power or signal loss. Using the thin filmarchitecture described above, permits selective positioning of fibers intight spaces and around objects or obstacles without optical power lossor signal loss. By having the bare or coated fibers placed on a thinfilm surface, it is also possible for the fiber connections to pass inthe thin space between the bottom of printed circuit boards top surfaceof a metal enclosure.

In instances where there are restrictions on actual placement of fibersdue to obstructions and other physical issues, placing fibers on a thinfilm 630 permits the route for each individual fiber and for the surfaceitself to be controlled so as to avoid obstacles, such as cutouts, screwmountings, support posts, low components, tall components, and otherobstructions. The fibers can be routed around these obstacles in orderto meet the bend radiuses and provide the connections between any twoendpoints.

In another embodiment, the Fiber Mesh Interconnect 10 uses bare orcoated fibers on a thin film 630 where the fibers can be physicallymated to the FC, SC, ST, LC, MPO, MXC, or other connectors intended forthe inside of the front or rear connector locations. These connectorscan be terminated, fusion spliced, or can be mated using othertermination process.

This method also permits increased fiber density in the area between thefront and rear connectors permitting additional connectors andconnectors with larger fiber counts on both the front and rear panels.

In another embodiment, the use of the Fiber Mesh Interconnect 10 canreduce the depth for an enclosure using standard cabling solutions.

In some applications of multifiber connections, the actual path lengthis important to ensure that one signal does not arrive before or afteranother signal in the same multifiber group. These are typically bondedsignal applications where the path length should be tightly matched. Inthis particular case, the individual fibers can be routed from oneconnector to another such that each fiber in the same multifiber grouphas the same fiber length regardless of the actual distance between theingress connector position and the egress connector position. Forexample, in one multifiber application, a path might be from oneconnector on the far left side of a panel to a connector on the farright side of the panel. At the same time a loopback connection may befrom one position to another position on the same multifiber connector.This would normally be either a very short loopback connection or alarge fiber route inside the enclosure which would occupy considerablespace and may bunch up fibers inside the enclosure potentially resultingin bend radius issues. By using the Fiber Mesh Interconnect 10 of thepresent application, fibers adhered to the substrate can result in acontrolled length, controlled bend radius, and fixed fiber routing pathin order to control the variability within fiber placement.

The Fiber Mesh Interconnect 10 of the present disclosure permits thecreation of a fiber interconnect scheme between a plurality of fiberoptic ports. In some embodiments, bundled fibers in variousconfigurations including ribbon fibers can be used in the Fiber MeshInterconnect 10. The individual or bundled fibers are adhered to a thinfilm, e.g., a Mylar sheet, using adhesives or other method to secure thefiber in place.

FIGS. 9 and 10 show one embodiment of a Fiber Mesh Interconnect 10 whereports or connectors 604 and 610 are interconnected by fibers 602 toprovide an interconnection between the plurality of fiber ports (orconnectors) 604 and 610. The fibers are terminated within the Fiber MeshInterconnect 10 by either single or duplex fiber connectors 604, such asFC, SC, ST, LC, or other single or duplex fiber optic connectors, or bymultifiber connectors 610, such as MPO and MXC connectors. A singlefiber connector 604 mates with an external equivalent connector type 608carrying a single or duplex fiber cable 606. Multifiber connectors 610mate with multifiber cables 612 terminated into multifiber connectors614.

In one embodiment, individual fiber optic fibers terminated using FC,SC, ST, LC, MPO, MXC, or other fiber optic connectors 604 and 610 can beconnected individually from point to point for each endpoint. In thiscase, the cross mapping of the endpoints is implemented on a perendpoint basis.

Another embodiment permits fiber optic cables using single fiberconnectors 604 which connect to single fiber cables 606 terminated insingle fiber connectors 608 such as FC, SC, ST, LC, or other singlefiber optic connectors to connect to an interconnect panel, which inturn provides the cross mapping in order to connect one end point to adifferent endpoint. This exemplary embodiment further simplifies thearchitecture since rather than have multiple individual cables, theinterconnect panel can support the cross mapping and use standardinstallation cables in the data center network.

In another embodiment, predefined fiber cable bundles comprisingmultiple fiber paths 602 can be constructed using the thin film 630connecting to connectors 604 and 610 using terminated FC, SC, ST, LC,MPO, MXC, or other fiber optic connectors 608 and 614 at the cable endswith the cross mapping of the configurations of the network devices in alocal interconnection scheme designed into the cable bundle. In thiscase, the interconnection scheme is simplified for the installer andreduces the possibility or cross mapping errors.

The individual ports can be FC, SC, ST, LC, MPO, MXC, or other types offiber optic connector 604 and 610. Thus, the Fiber Mesh Interconnect 10may be able to convert from one fiber connector type to anotherconnector type, so that the different fiber connector types may be mixedwithin the same system. In the case of multiple stranded fiberconnectors, such as MPO connectors 614, where a designated fiber isidentified by its position within the connector, the fiber mapping maybe from one position within the MPO to an identical position in adifferent MPO. In another variant, the fiber mapping may be from oneposition within the MPO to a different position in a different MPO. Inanother variant, the fiber mapping may be from one position within theMPO to a different position within the same MPO. In another variant, thefiber mapping may be from one position within the MPO to a differentposition in a different connector type, such as an FC, SC, ST, LC, MXCor other types of fiber optic connectors 608 and 614.

The individual fibers are placed onto the Mylar or other substratesurface either in groups or individually to create connections from onefiber endpoint position to a different fiber endpoint position.Individual fibers can be placed on a single row or layered over otherfibers such that the fiber mesh architecture becomes a three dimensionalstack of fibers. The individual fibers are then terminated onto an FC,SC, ST, LC, MPO, MXC, or other fiber connector types 608 and 614 asnoted above. Multi-position fibers such as MPO or MXC connectors 614,may have the individual fibers grouped and packed in ribbon strips forend terminations. The resulting arrangement produces a row of fiberoptic connectors interconnected by individual fiber strands to form theFiber Mesh Interconnect 10. As noted above, the Fiber Mesh Interconnect10 may be installed within a housing or enclosure, and in suchconfigurations, the connectors 604 and 610 could be arranged on a front,rear or side panel of the housing or enclosure. In one embodiment, thefiber optic connectors can all be arranged on the front panel. Inanother embodiment, the fiber optic connectors can all be arranged onthe rear panel. In another embodiment, the fiber optic connectors can bearranged with some connectors on the front panel and some connectors onthe rear panel. Likewise, there are applications where certainconnectors might be mounted on the top, bottom, or sides of the housingor enclosure.

In yet another embodiment, the connectors can be arranged in a verticalarrangement, such that the configuration results in a stacked set offiber optic connectors. Similarly, fiber connectors may exit theenclosure from any side of the enclosure depending upon the particularimplementation needed.

As noted, the Fiber Mesh Interconnect 10 may be positioned in a housingor enclosure with the fiber optic connectors on the outside of thehousing or enclosure. In such configurations, individual fiberconnections on the outside of the housing or enclosure have a dedicatedroute to another individual fiber connection. In this way, specificinterconnect and cross-connect patterns can be created within theenclosure and thus permitting the use of common off the shelf trunkcables and patch cables between one network device and another networkdevice or to multiple network devices in the case of multifiber cabling.

FIG. 10 shows one embodiment of a fiber mesh interconnect 10 housed in afiber mesh enclosure 11. Fiber mesh connectors 604 and 610 are coupledto external cable connectors 608 and 612 by fiber couplers 632 and 634.Fiber mesh connectors 604, external cable connectors 608, and fibercouplers 632 can be the type of single fiber connector type FC, SC, ST,LC or other single or duplex fiber type. Fiber mesh connector 610,external cable connectors 612 and fiber couplers 634 can be the type ofmultifiber connector type MPO, MXC, or other multi-fiber type.

The Fiber Mesh Interconnect 10 can include many variations. As definedabove, the implementation may be straight through from input port tooutput port utilizing either the same or different connector types andor connector sizes, or may have different input port to output portconnectivity.

In one embodiment, a Fiber Mesh Interconnect 10 can provide all theprimary path connections within a data center network. In anotherembodiment, a Fiber Mesh Interconnect 10 provides the primary andalternate path connections in a data center network. In a differentembodiment, a number of Fiber Mesh Interconnects 10 can coexist and orinterconnect to one another in the data center network.

It is also contemplated that a plurality of Fiber Mesh Interconnects 10may be in a single housing or enclosure, such that the connectors areaccessible through a one or more enclosure panels. In anotherembodiment, the plurality of Fiber Mesh Interconnects 10 within theenclosure may have connections from the different interconnects mixedtogether on multiple enclosure panels.

In another embodiment, the plurality of Fiber Mesh Interconnects 10within the enclosure may have connections mated internally from oneFiber Mesh Interconnect 10 to another Fiber Mesh Interconnect 10.

In yet another embodiment, the plurality of Fiber Mesh Interconnects 10within the enclosure may be switchable from Fiber Mesh Interconnect 10to a different Fiber Mesh Interconnect 10 in order to switch networkconfigurations. In this instance, a one enclosure with multiple FiberMesh Interconnect 10 panels, a mechanical or mechanized lever may removeone Fiber Mesh Interconnect 10 panel from the inside of the externalconnector ports and insert another Fiber Mesh Interconnect 10 panel intothe inside of the external connector ports. This permits reconfigurationof the fiber mesh network without re-cabling the external connections.

In another embodiment, an enclosure with multiple Fiber MeshInterconnects 10 may have the connections brought out from a singleinterconnect and a motor may, under the control of a controller, moveone Fiber Mesh Interconnect 10 enclosure from the internal connectorsand insert another Fiber Mesh Interconnect 10 enclosure into theinternal connectors of the second Fiber Mesh Interconnect 10.

The Fiber Mesh Interconnect 10 can have many different implementationsdepending upon the network size and topology. In one embodiment, theFiber Mesh Interconnects 10 can be placed on a hot insertable blade,which can be swapped in the field. In another application, the FiberMesh Interconnect 10 can be swapped in the field by replacing a damagedinterconnect substrate with a working interconnect substrate. In anotherexample, one Fiber Mesh Interconnect 10 implementation can be swappedfor a different Fiber Mesh Interconnect 10 wiring configuration.

Continuing to refer to FIG. 10, to take advantage of the manyimplementations of the Fiber Mesh Interconnect 10, each Fiber MeshInterconnect 10 is associated with a unique identifier in a givennetwork, and each physical port (or connector) and each fiber strand isassociated with a unique configuration implementation for thatparticular Fiber Mesh Interconnect 10. The orchestration system 400 candiscover these identifiers during a discovery cycle. The informationdiscovered by the orchestration system 400 includes the part number,fiber mesh configuration number, serial number, date of manufacture, andother relevant information. Interconnection information regarding fiberconnector types, fiber types and other information may be included fromthe Fiber Mesh Interconnect 10 itself or may be obtained by looking upthe information in an external database.

Techniques exist for identifying printed wiring boards and cables bysoftware capable reading a defined hardware object on the applicationwhich may include patterns of readable lines (bar codes), resistorvalues and positions to identify unique readable numbers, softwarereadable registers or other mechanisms capable of holding uniqueinformation. The Fiber Mesh Interconnect 10 can be equipped with one ofthese methods such that it is discoverable and readable by theorchestrations system 400.

The Fiber Mesh Interconnect Information 626 can be also implemented in avariety of different concepts such as printed on a bar code or datacode, such as a QR code and read by a bar code reader, QR scanner orother equivalent device. In a different embodiment, the Fiber MeshInterconnect Information 626 could be stored in an electronic memorycircuit such as a PROM, ROM, or register field, or other type of devicewhich can be read by an identification interface 628, such as a serialport, USB port, Ethernet port, or other means to read the device andelectronically pass the information read to a managing or monitoringentity.

The Fiber Mesh Interconnect Identification 626 information can be readby the orchestration system 400 through Identification Interface port628. In another embodiment, the Fiber Mesh Interconnect 10 may have aControl Processor on the Fiber Mesh Enclosure 11 assembly which may readthe Fiber Mesh Interconnect Information 626 and transmit it toorchestration system 400.

FIG. 11 shows the addition of physical identification technologies, suchas ninth wire technologies, RFID tagging, Connection PointIdentification (CPID), and other technologies, on the Fiber MeshInterconnect 10 couplers 704 and 710. Each Fiber Mesh Interconnectcoupler 704 or 710 will have the capability to determine the cablepresence and cable information available to Fiber Mesh Interconnect 10depending upon the information provided from the intelligent cable. Thisinformation is collected by a Media Reading Interface 718 in IntelligentFiber Mesh Enclosure 21 through intelligent media interface 702 andpassed to the CPU 720. The CPU 720 then reports the information to theorchestration system 400 via Fiber Mesh Interconnect Port 722.

In one embodiment, the Fiber Mesh Interconnect 20 may be designed withninth wire technologies interfaces. In another embodiment, the FiberMesh Interconnect 20 may be designed with RFID tagging technologyinterfaces. In another embodiment, the Fiber Mesh Interconnect 20 may bedesigned with CPID technology interfaces. In another embodiment, theFiber Mesh Interconnect 20 may be designed with other managed cableintelligence technologies. In another embodiment, the Fiber MeshInterconnect 20 may be designed with one or more of these differenttechnology interfaces in order to provide the capabilities of supportingmore than one particular managed intelligence technology in anapplication. This application may have the different technologiesseparate in the same assembly or may be used to bridge interfaces ofdifferent intelligence technologies to each other for example. Thisintelligent capability permits the orchestration system 400 to be ableto identify each cable connection connected to the Fiber MeshInterconnect 20. FIGS. 12-17 show an implementation according to thepresent disclosure of a TOR Fiber Mesh and an EOR Fiber Mesh.

Referring to FIG. 18, a Fiber Mesh Interconnect Expansion 30 accordingto the present disclosure simplifies the cabling within the data centernetwork and reduces insertion loss associated with multiple connectionsby using extensions in the fiber mesh interconnect expansion 30 toprovide dedicated connections directly to the devices.

As noted above, a Fiber Mesh Interconnect 10 terminates all the fiberson the Fiber Mesh Interconnect 10 into single fiber connectors 604 ormultifiber connectors 610. A fiber patch cable then connects the FiberMesh Interconnect 10 from connector 604 or 610 to a network device,e.g., a server or storage device.

FIG. 18 shows one embodiment of the Fiber Mesh Interconnect Expansion30, which is similar to the Fiber Mesh Interconnect 10 of FIG. 9, butalso includes Fiber Mesh Expansion cables 624. Each Fiber Mesh Expansioncable 624 extends off the edge of Fiber Mesh Interconnect 30. The FiberMesh Expansion is made by placing fibers 602 on the thin film substratefrom one connector 604, 610, 608, 612 to another connector 604, 610,608, 612. In the Fiber Mesh Interconnect Expansion 30, the Fiber MeshExpansion cables 624 are placed in the same manner on the Fiber MeshInterconnect 10 except that the fibers 602 extend off the thin filmsubstrate outside the physical enclosure and are terminated inconnectors 608 or 612 at some distance from the enclosure.

The fibers may be terminated to a connector 632 as a single fiber 602 ormay be terminated in a connector 634 which can support multiple fibers602.

The fibers 602 exiting the Fiber Mesh Interconnect Expansion 30 may alsobe encased in a sheathing 626 intended to protect the fibers from damageas they are routed to their intended destination.

In one embodiment as shown in FIG. 18, the Fiber Mesh InterconnectExpansion 30 may be implemented without an enclosure. In embodiment asshown in FIG. 19, the Fiber Mesh Interconnect Expansion 30 may beimplemented within Fiber Mesh Interconnect Expansion Enclosure 31. Thefiber extensions 624 exit the enclosure 31 through opening 622 which mayor may not have some form of strain relief to anchor the fiber or toprovide necessary strain relief against such hazards as minimum bendradius.

In one embodiment, all cables 624 from the enclosure 31 have the samefixed length. In another embodiment, cables 624 may have differentlengths depending upon the application. In one embodiment, the FiberMesh Interconnect Expansion 30 is located at the top of a rack. In thisembodiment the cables 624 are fed down the sides of the rack makingconnections to the servers, storage devices or both depending upon theimplementation. In another embodiment, the Fiber Mesh InterconnectExpansion 30 may be located at the end of a row of cabinets and thecable extensions 624 fan out to each rack in a row.

Preferably, each fiber has a predetermined length based on a givennetwork configuration and therefore, the Fiber Mesh Interconnect 10 canbe made as a Fiber Mesh Expansion 30 with the internal fibers extendedto the desired length and terminated at in factory. The completed FiberMesh Interconnect Expansion 30 assembly can then be installed at thecustomer with the cabling already routed in place.

The Fiber Mesh Interconnect Expansion 30 has individual cables 624 forthe intended endpoints which can be terminated with different connectors632 and 634 such as FC, SC, ST, LC, MPO, MXC, or other connector typesdepending upon the breakout requirement. The Fiber Mesh InterconnectExpansion cables 624 for the intended endpoints can also be terminatedwith intelligence cable connectors 708 and 714.

Similar to FIG. 11, the Fiber Mesh Interconnect Expansion connectors 636and 638 can be implemented with intelligent connectivity such as ninthwire technologies, RFID tagging, Connection Point Identification (CPID),and other technologies.

Another embodiment of the data center network of the present disclosureis the provision of a network device that supports the collection ofintelligent information from within the network device itself, thusimproving the accuracy of the readings and permitting direct reportingof the physical cable information to the orchestration system 400.

Each network device can report to the orchestration system 400 the typeof network device it is, e.g., a switch, server, storage device,interconnect panel, cross connect panel, along with relevant informationfor that network device, including number of ports, type of ports, speedof ports, and other physical information known to the network device.This information can be transmitted to the orchestration system 400.

Each network device also has a physical location within a data centersuch as in a particular rack in a particular row. This information iseither programmed into the network device which can be transmitted tothe orchestration system 400 or is entered directly into theorchestration system 400.

Another embodiment of the data center network of the present disclosureis the provision of an intelligent network device where managedintelligence connectors are incorporated into the intelligent networkdevice. By implementing intelligent network devices in the data centernetwork, the orchestration system 400 can collect not only the physicalinformation of the intelligent network device, but also each intelligentnetwork device can detect the insertion and removal of cables in thenetwork device connectors and can collect cable parameter information ofthe cables connected to the intelligent network device. The intelligentnetwork device can then report this information to the orchestrationsystem 400, which can map out each connection in the network.

The cable information provided to the orchestration system 400 mayinclude for each cable connection, the cable type, cable configuration,cable length, cable part number, cable serial number, and otherinformation available to be read by the Media Reading Interface 702.This information is collected by the Media Reading Device 718 and passedto the CPU 720 which in turn forwards the information to orchestrationsystem 400. With this information, the orchestration system 400 canidentify each unique cable within the data center network and know theend physical locations (including the geographic location) of each cableend as reported by each network Pathway Controlling Device 130.

With this information, the orchestration system 400 can determine eachsegment connection of the cable connection. With this information, theorchestration system 400 can determine the end-to-end connectivity forevery connection within the network.

For troubleshooting and maintenance, the orchestration system 400 canisolate connectivity down to a per port and per cable connector. Withthis information, the orchestration system 400 can identify which end ofa cable has been disconnected in most segments.

Additionally, because the orchestration system 400 has the end-to-endconnection information from the physical layer, Layer 2 protocolsincluding STP, ARP, and other discovery protocols are not needed fordetermining interconnections within the data center network. Rather thanthe Pathway Controlling Devices 130 trying to determine theirinterconnections, the orchestration system 400 can instead map out theinterconnections and program the routing tables into the PathwayControlling Devices 130.

Additionally, with this information, the orchestration system 400 canmake deterministic decisions on how to route traffic through thenetwork. The path may be selectable by overall connection length,individual segment length, port speed, number of interconnections in agiven path, physical security of a particular link, or other attributesthat may determine particular path selection.

The orchestration system 400 has the capability to display thisinformation in tabular, graphical, or other forms to a user.

The orchestration system 400 has the capability to collect and displayinformation changes in real time as they occur.

Referring to FIG. 20, a Network Interface Card (NIC) 80 within a server104 contains a switch 810 on the card where each switch port within theswitch 810 has the capability to interconnect any of the input ports 818in the set of ports 820 to any of the output ports 818 in the set ofports 820 where the set of ports 820 is limited only by the switch 810size and connectors installed on the NIC 80.

In one embodiment, this switch 810 and port connectors can be built intoa server main board (not shown). In another embodiment, the circuitrymay be part of a plug in card to the server 104. In any embodiment, thecapability allows the switch 810 on the NIC Card 80 to be able totransfer data between the server itself via the PCI interface connector812 and any single port 818 on the switch interface connector. Thecapability also exists to allow the server to transfer data betweenitself via the PCI interface connector 812 and multiple ports 818simultaneously as part of a multicast, broadcast, or other similarmultiport transfer mechanism.

The capability of a switch 810 within the server also permits the switch810 to receive data from one ingress port 818 and transfer it out to asecondary port 818 under the control of the switch 810 without involvingthe CPU or packet processing logic of the server. Likewise, the switch810 can receive data from one ingress port 818 and transfer it out totwo or more secondary ports 818 as part of a multicast, broadcast, orother similar multiport transfer mechanisms under the control of theswitch 810 without involving the CPU or packet processing logic of theserver.

FIG. 21 is an exemplary embodiment of a small, server based data centernetwork utilizing servers with onboard switch NICs 800. Each server 850can be connected to one or more of the other servers 850 by networkconnection paths 856. The number of connection paths 856 per server aredetermined by the size of the switch logic 810, number of ports in thetransceiver 804, and the connectors 802 on the NIC card 800. In thisarrangement, the CPU in each server 850 can communicate directly to anyserver 850 which has a direct connection path 856 between the servers.The CPU in each server 850 can communicate directly to any server 850where there is no direct connection path 856 between the two servers 850by sending the packets to a server 850 to which it has a directconnection path 856 which in turn will forward the packets to thedestination server 850. For example, server 850A could send a packet toserver 850F which in turn would forward the packet to server 850B. In analternate configuration the intermediary server 850F programs a directconnection within the switch logic 810 of the intermediary server 850F.Server 850A can then communicate directly to server 850B via theconnection set up in server 850F.

In instances where a direct connection creates an input port to outputport connection within switch 810, the server CPU is not needed toforward the data stream between the input port and output port. Thispermits server 850A to create a protocol independent data stream orencrypted data stream and send it directly to server 850F.

In another embodiment, a network can also include storage devicesequipped with NIC 800.

In another embodiment, a small network can be expanded by connectingsome of the direct connection paths 856 to Fiber Mesh Interconnects 852or other aggregation methods which in turn couple the data streams to anend of row aggregation or other switch.

Furthermore, the architecture permits this server switch logic 810 toconnect to traditional switch products in order to create connections tolarger network endpoints. The protocols supported in the switch 810 mayinvolve Ethernet, fiber channel, or other protocols. The connectors 802can include copper interfaces, such as Cat 5, Cat 6, Cat 7, other RJ45implementation variations, Fiber channel interfaces, optical interfacesincluding but not limited to FC, SC, ST, LC, MPO, MXC type connections.

The NIC 80 may have LEDs to indicate the port status of each individualport and LEDs for the state of the overall device. The LED blink patternwill be defined for each application. The LED color or colors may alsobe defined to indicate certain conditions. The NIC 80 may have an LCDdisplay on the enclosure to indicate the status of each individual port818 and/or the state of the overall device.

Another improvement of the data center network of the present disclosureis the provision of a NIC 80 to support the capability of obtainingintelligent information from within the NIC 80 itself, thus improvingthe accuracy of the readings and permitting direct reporting of thephysical cable information to the managing software. In this embodiment,the connectors can include copper interfaces, such as Cat 5, Cat 6, Cat7 other RJ45 implementation variations, Fiber channel interfaces,optical interfaces including but not limited to SC, ST, FC, LC, MPO, MXCtype connections.

Referring to FIG. 22, the architecture of the present disclosure alsopermits the implementation of the capability to interpret cableinformation from cables connected to the NIC 82, by obtainingintelligent information from within the cables. In addition tointerfacing to standard cables 612, adapter 832 has the capability, viainterface 834, to detect the presence of a cable connector 612 or 712inserted into intelligent adapter 832, and in the case of intelligenceequipped cable connector 714, read specific cable information by readingthe information in cable media 716. To ascertain cable information, theNIC 82 may be designed with ninth wire technologies interfaces, RFIDtagging technology interfaces, connection point ID (CPID) technologyinterfaces, or other cable managed intelligence technologies. In anotherembodiment, the NIC 82 may be designed with one or more of thesedifferent technology interfaces in order to provide the capabilities ofsupporting more than one particular managed intelligent technology.

Each NIC 82 equipped with intelligent cable interfaces has thecapability to determine the cable presence and/or cable informationavailable to the interface depending upon the information provided fromthe intelligent cable. In this embodiment, Media Reading Interface 836can read the physical cable information obtained from media interface716 on cable connector 714 and report this information to theorchestration system 400 via the main board CPU (not shown).

The cable information read from media interface adapter 716 via mediainterface 834 by media reading interface 836 and provided to the mainboard CPU may include for each cable connection of the cable type, cableconfiguration, cable length, cable part number, cable serial number, andother information available to be read by media reading interface logic836. This information is collected by media reading interface logic 836and passed to the CPU via PCI Interface 814 over PCI Interface Bus 816.The CPU then reports the information to orchestration system 400.Orchestration System 400 can use this information along with informationreceived from other data center network devices to map out theend-to-end connection paths of each cable connected in the data center.

The orchestration system 400 implements a method which providesend-to-end information regarding the overall path and the intermediaryconnections which make up and end-to-end path.

The orchestration system 400 collects the physical layer intelligentmanaged connectivity data from each switch, server, storage devices,interconnect panel, cross connect panel, and other devices in thenetwork which have managed interconnect capabilities.

A new High Density Pathway Controlling Device 60 is defined as shown inFIG. 23 with built in multiport transceiver modules 904 inside the HighDensity Pathway Controlling Device 60 rather than the SFF cages on theexterior of the device where SFP, SFP+, QSFP, or other modules can beplugged into the High Density Pathway Controlling Device 60. Theintention is to significantly increase the density of a switch orrouter.

The small footprint of multiport transceivers 904 allows multipletransceivers 904 within the High Density Pathway Controlling Device 60to increase the physical number of connections within the High DensityPathway Controlling Device 60 more than a standard switch or router withSFF module cages.

In another embodiment, the use of the multiport transceivers 904 permitsa smaller device physical size due the elimination of the spacerequirements necessary for a similar port density switch incorporatingSFF module cages.

By incorporating denser transceiver modules 904 inside the High DensityPathway Controlling Device 60, the number of connections per moduleincreases. Furthermore, the placement of the transceiver modules 904inside the High Density Pathway Controlling Device 60 can be staggeredwith respect to each module in order to more tightly pack the modulesinside the device.

The second aspect of the High Density Pathway Controlling Device 60 isto introduce the use of high density fiber connectors such as MPO, MXC,and other connectors 914 which have a high fiber count and smallfootprint. This permits effective use of the panel space for the moduleconnections inside the High Density Pathway Controlling Device 60.

FIG. 24 shows the addition of physical identification technologies suchas ninth wire technologies, RFID tagging, Connection PointIdentification (CPID), and other technologies on the High DensityPathway Controlling Device 70. Each High Density Pathway ControllingDevice 70 has the capability to determine the cable presence and orcable information available to the interface depending upon theinformation provided from the intelligent cable. This information iscollected by the Media Reading Device 906 and passed to the CPU 912. TheCPU 912 then reports the information via the Fiber Mesh InterconnectPort 922 to the orchestration system 400.

In one embodiment, the High Density Pathway Controlling Device 70 may bedesigned with ninth wire technologies interfaces. In another embodiment,the High Density Pathway Controlling Device 70 may be designed with RFIDtagging technology interfaces. In another embodiment, the High DensityPathway Controlling Device 70 may be designed with CPID technologyinterfaces. In another embodiment, the High Density Pathway ControllingDevice 70 may be designed with other managed cable intelligencetechnologies. In another embodiment, the High Density PathwayControlling Device 70 may be designed with one or more of thesedifferent technology interfaces in order to provide the capabilities ofsupporting more than one particular managed intelligence technology inan application. This application may have the different technologiesseparate in the same assembly or may be used to bridge interfaces ofdifferent intelligence technologies to each other for example.

This capability permits the orchestration system 400 to be able toidentify each cable connection connected to the High Density PathwayControlling Device 70.

Another improvement of the data center network of the present disclosureis to dynamically map fibers 918 in a configuration where all the fibers920 within a connector can be utilized, and at the same time providemulti-rate communications capabilities within the same connector. Theconcept that 10 Gbps ports may migrate to 40 Gbps ports and/or to 100Gbps ports is achievable by the bonding of fibers together to formmultifiber connections between endpoints. The 40 Gbps bandwidth isachieved by running four fibers in one direction for the 40 GbpsTransmit path and four fibers in the other direction for the 40 GbpsReceive path. Similarly, the 100 Gbps bandwidth is achieved by running10 fibers in one direction for the 100 Gbps Transmit path and 10 fibersin the other direction for the 100 Gbps Receive path. The current IEEE802.3 proposed implementation for these schemes is to use eight fibers(four transmit and four receive fibers) in a 12 fiber MPO for 40 Gbpsconnection. This means four fibers are wasted in this implementationscheme. For 100 Gbps communications, there are two implementationschemes. One uses 10 fibers out of 12 in a 12 fiber MPO with theremaining 2 fibers not used in the transmit path plus 10 fibers out of12 in a 12 fiber MPO with the remaining two fibers not used in thereceive direction. The other implementation scheme uses 10 fibers fortransmit plus 10 fibers for receive with four fibers unused in a 24fiber MPO. In these cases, migrating from a connection comprising onlyof 10 Gbps connections to 40 Gbps or 100 Gbps requires bothreconfiguring the fiber transmit and receive connections inside theconnectors and also the loss of use of some of the fibers in theconnector.

The data center network according to the present disclosure permits thedynamic mapping of fibers 918 to a configuration where all the fibers920 can be used within a connector, and at the same time providemulti-rate communications capabilities within the same connector. Animproved implementation scheme is to utilize all the fibers 920 withinthe connector and allow the interconnect panels and switches to separatethe individual links 918 from the bonded links. This also permitsexpansion of 12 fiber MPO configurations to 24, 48, 72, or other MPOfiber combinations in order to be able to support multi-rate andmultifiber applications in the same connector.

This also permits expansion of 12 fiber MPO configurations to MXC orother high fiber count connectors 612 or 712 without the requirements ofpredefined bonding configurations for multifiber applications in thesame connector.

In a different embodiment, single transmission connections such as 1Gbps, 25 Gbps, 56 Gbps speeds, or other speeds may be intermixed in thesame MPO or MXC or other high fiber connector with CWDM, DWDM, and othermulticolored fiber transmission schemes.

Referring now to FIGS. 25-32, the orchestration system 400 is describedin more detail. The orchestration system 400 is similar to a networkmanagement system that includes conventional processes, such as NetworkTopology, Routing, Alarm, Security, Performance, Audit Trails, ProjectManagement, Inventory, and other processes, as shown in FIG. 25. Inaddition, the orchestration system 400 of the present disclosureincludes a number of network management functions that conventionalmanagement systems do not have, including discovery of the physicalinfrastructure of the data center network, determining the physicaltopology of the data center network, tracking physical network devicesand other network components, and providing definable network paths.

In one embodiment, the functions of the orchestration system 400 of thepresent disclosure can be set forth as, planning functions,initialization functions, and operation functions. Planning functionsallow users to architect the physical layout of the data center networkwithout physically being at the site. Initialization functions helpnetwork device deployment processes perform much quicker thantraditional processes. With the initialization functions, theorchestration system 400 can do initial configuration in minutes whichis much faster than the hours needed for conventional initialconfiguration. Operation functions provide element configuration,monitoring, diagnostics, tracking, and network management.

Planning is done via a three dimensional planning application (3DPlanner) 502. With the 3D Planner 502, a designer can architect theirnetwork infrastructure by defining the building, datacenter, zones,rows, racks, rack network devices, modules, port and cable types viadragging and dropping components from the toolbar. Components in the 3DPlanner 502 are called containers. Each container is associated with aunique identification which is used to determine its identity andaddress. The 3D elevation of racks and rack units provide realisticvisualization and identification. The 3D Planner 502 also provides atemplate for faster and easier replication of existing configurations.

The 3D Planner 502 can be incorporated into the orchestration system 400or it can be a standalone client that communicates with orchestrationsystem 400. The 3D Planner 502 screen layout can be implemented in manydifferent arrangements. In one embodiment, 3D Planner 502 screen layouthas a tool bar on top and component bar at the left side. The mainscreen is where devices and the data center area are shown. The firstview of the main screen is a map view where the screen displays iconsrepresenting the data center buildings. The toolbar is similar to otherapplications, which contain buttons for easy access to functions, suchas “save”, “delete”, “export”, and other configuration commands. Thecomponent bar contains multiple tabs; each tab contains a group ofcomponents. The component bar has a building tab containing icons todefine buildings; the data center tab contains icons to define a datacenter within the building; the zone tab contains icons to define a zonewithin data center, the row tab contains icons to define a row withinthe zone; the rack tab contains various racks for creating racks withinthe row; the rack unit tab contains different models and types of racknetwork devices, such as servers, switches, and other devices, as wellas different types of patch panels; the module tab contains variousmodels of modules (blades) that can be added to a rack unit space; theport tab allows the user to add ports to the device; the harness andcable tab allows the user to add cabling. These tabs also contain iconsrepresenting template configurations. In other embodiments, the 3DPlanner 502 may have different arrangements or layouts of tool bars,component bars, icons, and other layout differences.

In general, the user can define a building, data center, zone, row,rack, rack unit, module, port, harness and cable by dragging componenticons in the component bar over to the main view and dropping the iconsat user selected locations. In one embodiment, once the user drops theicon on the main view, usually a pop-up dialog appears asking the userto enter information (such as dimension, IP address, name, descriptionand so on). Once the user clicks “apply”, a message is sent toorchestration system 400. The orchestration system 400 receives a“create” message and creates the component, and then sends back anacknowledgement. When the 3D Planner 502 receives the acknowledgementfrom the orchestration system 400 with the information provided, itdraws the component in the main view as a visual acknowledgement to theuser. The components are drawn in 3D as if the user is looking at theactual physical structure. The same procedure applies to all components.To add a rack unit, the user double clicks on the rack to bring it intoedit mode, in which user can drag and drop the rack units on to therack. The same basic procedure is followed when the user wants to addmodules onto rack unit. In other embodiments, the process can beimplemented in different steps or techniques to achieve the sameobjectives.

Adding cables between network devices can be done in several ways suchas 3D Planner 502 network tree views, bundle cable views, or point ofuse view. When using network tree views, clicking a button in thetoolbar brings up a tree form dialog which has source and destinationtrees. In this dialog, the user can expand the tree to select a port onone tree and drag and drop that port over to another port on the othertree to connect the two ports together. Once dropped, a “confirm”message is shown. Once the “apply” button is pressed, a message is sentto the orchestration system 400 requesting to make the connection fromthe selected port. The orchestration system 400 grants that request andsends back an acknowledgement which triggers a completion indication atthe GUI side.

Referring now to FIG. 26 which shows a flow diagram for the cableverification process 530 for the cable assignment process. When addingsingle cables or bundled cables to the 3D Planner 502 topology view, thecables can be dragged and dropped onto the rack. A single cable may be asimplex or duplex cable intended to provide one port connection at eachend of the fiber and/or copper wiring cable. A bundled cable is apredefined collection of fibers and/or copper wiring made to fit certainrack configurations. A bundled cable may have multiple fibers or copperwires with a single connection at either end or a cable with multipleconnectors at one or both ends in the case of break out cables. When asingle or bundled cable is dropped in place, the association betweencable connection and device ports are made. Cables can also be added inthe device view where each port is visible. This point of use methodallows the user to perform a mouse click on the ports to bring up adialog for the user to select the destination port. Before accepting acable connection, a series of verifications such as connectorcompatibility, cable length, port compatibility, and device locations isexecuted by the 3D Planner 502 to ensure the configuration is compatiblewith the intended path connection.

Once a certain plan configuration is completed, the plan can be saved asa template so that it can be replicated quickly and easily. When theuser clicks on a button to save the plan as a template, a request issent to the orchestration system 400, the orchestration system 400process saves the template and sends it back to the 3D Planner 502. The3D Planner 502 receives the acknowledgement and then draws an iconrepresenting the saved template in the component bar.

After the planning phase and the installation of the components iscompleted, the initialization process can be carried out. The componentscan be already configured devices from inventory or un-configureddevices sent from a manufacturer. Initialization is done via the 3DInitializer 504 which instructs the user (e.g., a technician) step bystep to configure the network devices.

The 3D Initializer 504 may be incorporated into the orchestration system400 or it may be a standalone client application that communicates withorchestration system 400. When starting up, the 3D Initializer 504 logsin and retrieves information from the orchestration system 400. Oncereceiving a valid response from the orchestration system 400, it willdraw the rack, rack units, and other components similar to the layout inthe 3D Planner 502. The user selects a device in a rack and clicks the“configure” button. A dialog box will pop up with instructions for theuser to follow. Different types of instructions may be provideddepending on the type of device to be initialized. In general theprocess is as simple as selecting a device, plugging the cable in to thedevice as directed by the initializer, clicking the “configure” button,waiting until the process is complete, and then moving to the nextdevice in the view. During the configuration process, the 3D Initializer504 retrieves information that was entered during the planning processand selectively picks the information to send down to the device.

Once the initialization process 504 is complete, the system is in anoperational ready state with basic functionality. The orchestrationsystem 400 operation functions provides additional functionality to theorchestration system 400, including data flow management, definition andidentification, track-able and monitor-able physical connections,physical path discovery, segment disconnect detection, and bit errorrate detection.

In operation, the orchestration system 400 discovers the network usingthe discovery process 506 algorithm, which is described below and shownin FIG. 27. The discovery process 506 includes two operations: acollection process 508 and an association process 509. The collectionprocess 508 is carried out by multiple controller modules to gather allnetwork device information, all port information, and all connectivityinformation between network devices. The association process 509correlates the network device information, port information, andconnectivity information between network devices to create relationshipsand connectivity between the devices in the data center network.

FIGS. 11, 22, and 24 show examples of different network devices andcables equipped with intelligent physical identification technologies,such as ninth wire technologies, RFID tagging, Connection PointIdentification (CPID), and other technologies which are used to identifycable related information at each port in a network device. Dependingupon the technology implemented in the device, the device may be able toascertain the presence of a cable, the type of cable, the length of thecable, make of cable, serial number of the cable, and other physicalinformation available to be determined from the device port. Knowing theinformation of both ends of a cable from different device ports, theassociation process 509 can identify a physical cable connection andbetween two devices and therefore can identify a cable as a knownphysical cable between two devices. With the physical cable information,the association process 509 uses algorithms to identify cableconnections and associate unidentifiable cables with device portconnections.

Each controller also associates connectivity information to createconnections between network devices within its coverage. All informationis then sent to a central association process 509 module to finish theassociation. The results are presentable in graphical form, similar tonetwork diagram shown in FIG. 28.

Physical layer cabling additions: In an exemplary embodiment, theorchestration system 400 can display newly added cabling on a physicaltopology view of the network, or updated by manual processes, guided byprompts from the orchestration system 400 alerting the operator byhighlighting, flashing, or blinking colors, symbols, or text on top ofaffected devices. Cabling will appear on the graphical representation ofthe network topology, with, in one embodiment, blinking yellow dots ontop of the either end of the cable, prompting the user to acknowledgethe new cable. In other embodiments, different visual indications can beused to represent the same scenarios as detailed here.

Physical layer cabling removal: When a cable is physically removed fromthe network, the cable remains in the topology view; however the colorof the cable is changed to red, as well as the devices attached toeither end of the cable. The cable is also identified by red blinkingdots at both ends of the cable, prompting the operator to acknowledgethe change in topology.

If the cable that was removed is added back into the network in the sameposition, the cable and attached devices change back to their defaultcolors. The cable is displayed in the network topology with yellowblinking dots, prompting the user to acknowledge the change at bothends.

If a different or replacement cable is added back into the network inthe same position rather than the original cable, the cable and attacheddevices change back to their default colors. The cable is displayed inthe network topology with orange blinking dots, prompting the user toacknowledge the change at both ends.

Intrusion detection: In an exemplary embodiment, if one cable end isremoved and replaced with a different cable connection, a red blinkingdot is placed on the end of the cable that changed in the networktopology, while the other end which was not disconnected remains clear.The red blinking dot identifies that the cable end change was notauthorized.

After the discovery process 506 is completed, the orchestration system400 compares the discovered data center network with the planned networkto determine if there are any differences. The differences are presentedto the user for resolution decisions. The detection mechanism checks fordifferences in device information, connectivity, and cablecharacteristics. Also, once discovery process 506 is completed andvalidated, the orchestration system 400 then calculates all possiblepaths from one end device to another following the process set forth inFIG. 29 to ensure the path configurations do not violate the rulespertaining to the maximum number of routes, maximum length of the route,and maximum number of hops per connection.

As new devices are added to the network and as these devices areconnected to the orchestration system 400 via the management interface401 they are discoverable by the orchestration system 400 and the newlydiscovered devices will be displayed in the 3D Planner 502.

Given identifiable physical paths, the orchestration system 400discovers physical layer connectivity, the physical topology network,and the logical network. Physical topology is a network topology thatrepresents one or more physical devices connected to each other byphysical cables. Logical topology is a network topology that representsone or more physical and/or logical devices connected with each other byphysical cables and/or logical connections. Using the characteristics ofphysical connectivity in combination of the data link layer and higherlayers that provide logical connectivity, the orchestration system 400operations functions can traverse the network to find the missingphysical connections. In order for the orchestration system 400 tocalculate data stream route paths and locate fault conditions, it firstidentifies each device and cable segment in the network.

FIGS. 30a-30c are exemplary network diagrams illustrating the capabilityof the orchestration system 400 discovery process during conditionswhere a particular cable in the network is not readily identifiable. Theorchestration system 400 first discovers the existence of networkdevices and then the orchestration system 400 associates all ports basedon cable identification number to create connections 516 between thenetwork devices. FIG. 30a indicates a scenario where all connections(also referred to as “links”) 516 are identifiable.

FIG. 30b shows one possible scenario where a connection 518 between hostA 510A and host B 510B is unidentifiable at the edge of the network, andthe unidentifiable connection 518 in this exemplary scenario is theconnection between switch B 512B and host B 510B. The host B 510B can beany network device. In this situation, the discovery process discovers apath exists between host A 510A and host B 510B, and also that allconnections 516 are identifiable except the unidentifiable connection518. In order to recognize the unidentifiable connection 518, theorchestration system 400 can determine the connectivity using a numberof different methods, including: a) a Layer 2 and above connectivitymethod, b) a path traversal method using a route calculation method,and/or c) the fact that there is only one path capable of reaching hostB 510B from switch B 510B. In FIG. 30b , using Layer 2 and aboveconnectivity method indicates that there is at least one path going fromthe switch A 512A to host B 510B. Using a path traversal process, theorchestration system 400 determines the available paths from switch A512A through the patch panels 514 to host B 510B. From the outgoing portof the patch panel 514, the orchestration system 400 can reliably make adetermination that the outgoing port on switch B 512B is connected tohost B 510B via an unidentifiable cable, and can now classify this cablebased on the known connectivity information.

FIG. 30c shows another exemplary unidentifiable connection in the datacenter network, this time between patch panels. The discovery of theunidentifiable connection in FIG. 30c is similar to that of the processthat was done for the network type in FIG. 30b . In this situation, aconnection can be established from host A 510A to host B 510B overdifferent segment combinations. Since most of the segments areidentifiable by the orchestration system 400, the orchestration system400 can draw out the potential paths that a data stream can take fromhost A 510A to host B 510B. However with the unidentifiable path, theorchestration system 400 only knows that a cable has been connected topatch panel 514B and a cable has been connected to patch panel 514C. Theorchestration system 400 needs to know if these two endpoints arerelated as part of the same cable or if the connections are associatedwith different cables that connect to different endpoints. Theorchestration system 400 traverses the path from the switch 512A topatch panel 514C and then traverses a data stream through theunidentifiable cable port on patch panel 514C, and then monitors theother unidentifiable ports in the network to locate the other end of thecable which in this case terminates on patch panel 514B. Theorchestration system 400 now has connection knowledge of the previouslyunidentifiable cable. Similarly, the orchestration system can runsimilar path traversal mechanisms for other types of network devicesbesides patch panels.

The orchestration system 400 operation functions can also detectconnection tampering. If one of the connections in a managed data centernetwork is removed, added, or changed, the orchestration system 400 candetect the change of state and provide an indication of the tampering inreal time. Even if the connection is cut, the orchestration system 400is able to determine which cable is cut. FIG. 31 shows how devices areinterconnected across the network and is used as an example toillustrate the method of segment-disconnect detection. At the edges,data is sent to and from a single point 510 over a single cable 520 andthen aggregated in a switch 512 or patch panel 514 which combine pathsfrom multiple hosts 510 into bundled cables 522 where at the center ofthe data center network, data is sent to and from many intermediarypoints over bundled cables 522. Given this, the orchestration system 400considers connections near the center over bundled cables as connectionsthat can share data from many sources. Thus, the orchestration system400 concludes that connections at an edge affect only the edge and thedata going into that edge, whereas connections near the data centernetwork center affects many edges and their data. With thatrelationship, the orchestration system 400 can locate a brokenconnection using a relational algorithm. With the visibility of thenetwork, the orchestration system 400 uses the relational algorithm 524to look at the edges to see what data paths are broken and then traverseto the common point(s) of breakage. The relational algorithm 524 isrepresented in FIG. 32.

In addition, the orchestration system 400 discovery mechanisms canreverse the topology network into physical elevation structure, as in a3D rack elevation in the 3D Planner 502, and using the identificationnumbers that were assigned to the network devices at the time ofplanning.

With the orchestration system 400 operations functions, data paths canbe assigned or shown based on a particular data type, application,protocol, or end-to-end path route. For example, with a VLAN, a user canchoose to show where a certain VLAN will travel through the data centernetwork. Alternatively, the user can define a specific path as to how aparticular VLAN will travel through the data center network. Data pathscan be viewed by selecting a device, an application type, protocol type,or a flow in the topology graph. Upon selection, a highlighted path isshown in the topology graph. When a certain path is used to deliverspecific traffic, the user can choose endpoints and select one of thepaths for an available application type or protocol type. Assigned pathscan be viewed, changed, or removed.

The orchestration system 400 also allows setting up monitoring sessionsor ports via point and click in the topology map. Tapping is theduplication or splitting of data paths for routing the secondary pathtypically to a network monitoring device in order to performtroubleshooting, recording, logging, performance measuring and otherfunctions on the data stream. The created monitoring sessions andmonitoring ports are saved in the database which can be easily retrievedand managed.

End-to-End Server Encryption

The data center network according to the present disclosure is capableof providing a secure connection from server to server through the datacenter network. A secure path is dedicated to the server to serverconnection and is not available to any other network device in the datacenter network. Because the orchestration system 400 has knowledge ofall the paths and devices in the data center network, it can assignspecific paths through devices and enable a secure connection betweenthe two endpoints. The secure connection appears as a clear channelpath, where from the source server to a destination server, packets arenot processed, but merely forwarded bit by bit. This also enables thedevices at the connection endpoints to encrypt any part or all parts ofany PDU (Protocol Data Unit) type before transmission.

In addition to providing a clear channel path that enables transfer ofencrypted PDUs, the physical layer is secured as well through ConnectionPoint Identification (CPID) enabled cabling, CPID readers on panels,switches and every network device where CPID cables connect. All CPIDreaders feed connectivity information up to orchestration system 400.

Since the orchestration system 400 can determine the connectivity ofevery cable segment and intermediate network device and panel in a pathbetween two endpoints, the orchestration system 400 can determine ifthere are physical layer breaches in the network and has thecapabilities to isolate the breach down to a device or single cablesegment. Once a breach has been detected, the orchestration system 400can automatically disable data transmission from the endpoint deviceports as a means of stopping unauthorized tapping, monitoring, orrerouting of network data.

What is claimed is:
 1. A data center network, comprising: one or morerows, wherein each row has one or more racks, and wherein each of theone or more racks has at least one network device and at least onetop-of-rack network switch; and at least one end-of-row fiber meshinterconnect in communication with each top-of-rack network switchwithin the same row of the one or more rows, such that each top-of-racknetwork switch has a direct connection to every other top-of-racknetwork switch within the same row; wherein each top-of-rack networkswitch comprises: a housing having one or more connection panels; and aset of ports, wherein each port within the set of ports is configured toreceive data streams from at least one network device within each of theone or more racks, and to transmit data streams to at least one networkdevice within each of the one or more racks, wherein each port in theset of ports includes a connector and at least one transceiver opticallycoupled to the connector, and wherein the connector is mounted to theone or more connection panels for connecting to the at least one networkdevice and the end-of-row fiber mesh interconnect; and wherein the fibermesh interconnect, comprises: a housing having one or more connectionpanels, wherein each connection panel includes a plurality ofconnectors; and a plurality of individual optical fibers within thehousing and connected between one or more of the plurality of connectorsto provide a direct optical fiber connection between connectors, theplurality of individual optical fibers being adhered to a thin filmmedium and arranged on the thin film medium in a predefined mapping. 2.The data center network according to claim 1, further comprising anorchestration system that controls the flow of data streams between eachtop-of-rack switch and each end-of-row fiber mesh interconnect.
 3. Thedata center network according to claim 1, wherein the one or more rowsare arranged in double wide racks configured as even and odd rack pairs,and wherein the top-of-rack network switch has at least 42 ports toprovide connections to at least 42 network devices in each of the doublewide racks.
 4. The data center network according to claim 1, wherein theone or more rows are arranged in double wide racks configured as evenand odd rack pairs, and wherein the top-of-rack network switch has atleast 84 ports to provide connections to at least 84 network devices ineach of the double wide racks, and at least 24 port connections to endof row switches in the data center and at least 4 port connections toevery other data center network switch in each double wide rack in thedata center row.
 5. The data center network according to claim 1,wherein each port connector can be one of a CAT 6, CAT 6E, CAT 7, FC,SC, ST, LC, MPO, or MXC connector.
 6. The data center network accordingto claim 1, wherein one or more of the network devices are equipped witha physical identification system.
 7. The data center network accordingto claim 6, wherein the physical identification system comprises one ofninth wire technologies, RFID tagging, or connection pointidentification.
 8. A data center network fiber mesh interconnect device,comprising: a housing having one or more connection panels, wherein eachconnection panel includes a plurality of connectors; and a plurality ofindividual optical fibers within the housing and connected between oneor more of the plurality of connectors to provide a direct optical fiberconnection between connectors, the plurality of individual opticalfibers being adhered to a thin film medium and arranged on the thin filmmedium in a predefined mapping.
 9. The data center network fiber meshinterconnect device according to claim 8 wherein the plurality ofoptical fibers are adhered to one or more thin films.
 10. The datacenter network fiber mesh interconnect device according to claim 9,wherein the plurality of optical fibers can be routed on the thin filmin defined connection patterns between the connectors in a single layer.11. The data center network fiber mesh interconnect device according toclaim 9, wherein the plurality of optical fibers can be routed on thethin film in defined connection patterns between the connectorsoverlapping previously adhered fibers creating a multi-layer fiberinterconnection.
 12. The data center network fiber mesh interconnectdevice according to claim 8, wherein one or more of the plurality ofconnectors comprise multi-fiber connectors and one or more single fiberconnectors, and the predefined mapping between the connectors includes:connecting an optical fiber from one position within a multi-fiberconnector to a different position in a different multi-fiber connector;connecting an optical fiber from one position within a multi-fiberconnector to a different position within the same multi-fiber connector;connecting an optical fiber from one position within a multi-fiberconnector to a single fiber connector; or connecting an optical fiberfrom one single fiber connector to a different single fiber connector.13. The data center network fiber mesh interconnect device according toclaim 12, wherein the multi-fiber connectors comprise MPO or MXCconnectors, and the single fiber connectors comprise FC, SC, ST and LCconnectors.
 14. The data center network fiber mesh interconnect deviceaccording to claim 8, wherein one or more of the plurality of connectorscomprise multi-fiber connectors, and the predefined mapping between themulti-fiber connectors includes: connecting an optical fiber from oneposition within a multi-fiber connector to a different position in adifferent multi-fiber connector; or connecting an optical fiber from oneposition within a multi-fiber connector to a different position withinthe same multi-fiber connector.
 15. The data center network fiber meshinterconnect device according to claim 14, wherein the multi-fiberconnectors comprise MPO or MXC connectors.
 16. The data center networkfiber mesh interconnect device according to claim 8, wherein one or moreof the plurality of optical fibers extend through an opening within thehousing such that the one or more of the plurality of optical fibers areexternal to the housing to provide direct connections to externaldevices.
 17. The data center network fiber mesh interconnect deviceaccording to claim 16, wherein the external optical fibers are encasedin a protective sheathing to protect the optical fibers from damage. 18.A data center network, comprising: one or more rows, wherein each rowhas one or more racks, and wherein each of the one or more racks has atleast one network device and at least one top-of-rack fiber meshinterconnect; and at least one end-of-row fiber mesh aggregation incommunication with each top-of-rack fiber mesh interconnect within thesame row of the one or more rows, such that each top-of-rack fiber meshinterconnect has a direct connection to every other top-of-rack fibermesh interconnect within the same row; wherein each top-of-rack fibermesh interconnect comprises: a housing having one or more connectionpanels, wherein each connection panel includes a plurality ofconnectors; and a plurality of individual optical fibers within thehousing and connected between one or more of the plurality of connectorsto provide a direct optical fiber connection between connectors, theplurality of individual optical fibers being adhered to a thin filmmedium and arranged on the thin film medium in a predefined mapping. 19.The data center network according to claim 18, wherein each end-of-rowfiber mesh aggregation comprises: a housing having one or moreconnection panels, wherein each connection panel includes a plurality ofconnectors; and a plurality of optical fibers within the housing andconnected between one or more of the plurality of connectors in apredefined mapping to provide a direct optical fiber connection betweenconnectors.